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Introduction 


There is increasing recognition among educators, researchers, policymakers, and 
the broader public that students’ social-emotional learning (SEL) is a critical component 
of success for both academic and life outcomes,! and that schools can support students’ 
SEL through practices that improve schools’ culture and climate.*? Research also shows 
that classroom teachers, in particular, can play a critical role in establishing classroom and 
school environments that contribute to students’ social and emotional development.**° 


The increasing attention on educators’ impacts on students’ SEL is evident in the 
2015 Every Student Succeeds Act (ESSA), which requires that states measure at least one 
indicator of school quality or student success related to non-cognitive outcomes, such as 
measures of student engagement or school climate and safety. Similarly, in California, the 
Local Control Funding Formula (LCFF) and the related Local Control Accountability Plan 
(LCAP) process require districts to develop and report local measures of school culture 
and climate, such as surveys of students, parents, and teachers.® 


The growing interest—both in California and nationwide—in using student surveys to 
measure student progress and school quality raises an important question about whether 
such surveys can reliably distinguish the impacts that different educators and schools can 
have on non-cognitive dimensions of success. Much in the same way that many states 
and districts have prioritized assessing student growth in academic achievement—rather 
than merely assessing student proficiency or other measures of attainment—measuring 
student growth in non-cognitive skills might also be more informative and equitable than 
relying on attainment alone. Such growth measures could then serve to highlight effective 
classroom supports that are successfully moving the needle to improve students’ social- 
emotional development. 


In this brief, based on a longer working paper,’ we aim to build upon previous 
research showing that teachers have the potential to influence students’ non-cognitive 
skills? by examining whether student growth in self-reported SEL differs among classrooms 
within a school. We also extend our own prior work??° examining how much of students’ 
growth in SEL from one year to the next can be attributed to the school a student attends. 
In this brief, we use survey response data from more than 40,000 fifth-grade students 
across five large districts in California to estimate value-added models for four SEL 
constructs. We assess the predictive power of these models, compare the results of the 
models to value-added models for math and English language arts (ELA), and examine 
whether classrooms that have large impacts on students’ SEL also have large impacts on 
students’ academic growth. 
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There are five key findings from this work: 


1. A greater proportion of the variation in students’ self-reported SEL is explained 
across classrooms within a given school than across different schools, although 
most of the variability in SEL outcomes is explained at the individual student level 


2. Classrooms in our sample serve students with different starting points in SEL 
within schools, which suggests it is important to control for those differences in 
order to measure the impact of classrooms on students’ SEL over the course of a 
school year 


5. We can estimate classroom-level growth models to control for different starting 
points in students’ SEL, but these models have less predictive power than similar 
models of academic achievement and, as a result, they may not control for 
students’ starting points as well 


4. The variance of classroom-level impacts on students’ SEL outcomes extracted 
from these growth models is similar in magnitude to impacts on students’ math 
and ELA outcomes, but it is a smaller proportion of the total variance to explain 
compared to math or ELA 


5. Classrooms that have a large impact on students’ growth in math and ELA are 
not necessarily the same classrooms that have a large impact on their growth in 
SEL (or vice versa) 


Measuring SEL in the CORE Districts 


We use data from California’s CORE Districts, a consortium of eight school districts 
collectively serving more than one million students attending roughly 1,500 schools. The 
CORE Districts administer a survey of students’ SEL each spring to students in grades four 
through 12. Students are asked to self-report their responses on a five-point Likert scale 
to items assessing four SEL domains: growth mindset, self-efficacy, self-management, and 
social awareness. The surveys include between four and nine items for each of the four 
constructs, for a total of 25 items on the survey. 


The CORE Districts describe the four SEL domains as follows: 


¢ Growth mindset is the belief that one’s abilities can grow with effort. Students 
with a growth mindset see effort as necessary for Success, embrace challenges, 
learn from constructive criticism, and persist in the face of setbacks 


e Self-efficacy is the belief in one’s own ability to succeed in achieving an 
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outcome or reaching a goal. Self-efficacy reflects confidence in the ability to 
exert control over one’s motivation, behavior, and environment 


e Self-management is the ability to effectively regulate one’s emotions, thoughts, 
and behaviors in different situations. This includes managing stress, delaying 
gratification, motivating oneself, and setting and working toward personal and 
academic goals 


¢ Social awareness is the ability to take the perspective of and empathize with 
others from diverse backgrounds and cultures, to understand social and ethical 
norms for behavior, and to recognize family, school, and community resources 


In order to use the SEL survey to measure students’ growth in SEL from one year 
to the next, we transform students’ raw responses into scale scores estimated from a 
generalized partial credit model (GPCM),” a step we describe more fully in other work. 


The Development of an SEL Classroom Growth Model 


The sample in this brief includes approximately 44,000 fifth-grade students in 
5,622 classrooms at 724 schools that are located tn five participating CORE districts. We 
focus on students in fifth grade because these students are frequently in self-contained 
classrooms, and we limit our analysis dataset to those fifth-grade students who are linked 
to one and only one teacher for instruction in all observed subjects. Our classroom-level 
growth models use data from the survey administered to fourth graders in 2015-16 as the 
pretest measure of students’ SEL, and data from the survey administered to those same 
students as fifth graders in 2016-17 as the posttest measure of students’ SEL. In addition 
to the student SEL survey data, we use data from the Smarter Balanced Assessment 
Consortium (SBAC) math and ELA assessments from the spring of 2015-16 and of 2016- 
17, which enables us to compare students’ growth in SEL to their growth in math and ELA 
achievement. 


We first examined whether there are differences in students’ SEL outcomes (i.e., 
2016-17 survey responses), as well as in their prior levels of SEL (i.e., 2015-16 survey 
responses), and whether those differences are attributable to classrooms specifically, 
rather than schools more generally. To do so, we decompose the variance explained in 
the SEL posttest (2016-17) and in the SEL pretest (2015-16) at three levels: across-schools, 
across-classrooms-within-schools, and within-classrooms. This allows us to estimate the 
amount of variability in students’ SEL due to the school the student attends, the classroom 
the student is in, and additional unexplained student-level variance, respectively. 


We next applied a value-added model framework to construct classroom-level 
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growth measures of students’ SEL in order to assess whether there are differences in 
students’ growth in SEL from 2016 to 2017 that can be attributed to the classroom 

a student is in. We also estimated these models with outcome measures of ELA and 
math. To do so, we estimated six separate models that each predicted one outcome 
(i.e., the student's scale score for math, for ELA, or for one of the four SEL constructs) 
using measures of all six scale scores in the prior year, along with student demographic 
characteristics (English language learner status, disability status, economic disadvantage, 
homelessness, foster care, and race/ethnicity) and a fixed effect for the classroom the 
student was assigned to. We estimated this regression using an errors-in-variables (ElV) 
method that accounts for measurement error in the prior-year assessment measures. We 
used estimates of Cronbach's alpha for lagged SEL constructs, and conditional standard 
errors of measurement for SBAC scores. 


To gauge the magnitude of the estimated classroom-level impacts on students’ 
SEL, we applied the variance decomposition approach to a student-level measure of 
growth estimated in the value-added regression, so that we could separately investigate 
the contributions of school and classroom effects to students’ growth in SEL. Finally, we 
computed correlations among the classroom effects estimated from the six value-added 
models in order to examine whether classrooms where students show high growth in the 
four SEL constructs also show high growth in academic outcomes. 


Findings 


Finding 1. A greater proportion of the variation in students’ self-reported SEL outcomes 
is explained across classrooms within a school than across different schools, 
although most of the variability is explained at the student level. 


Table 1 presents a decomposition of the variance in the posttest, showing the share 
of variance explained at three different levels: (1) across schools, (2) across classrooms 
within schools, and (3) within classrooms (i.e€., across students within classrooms). We 
can see in Table 1 that the magnitude of the differences across-classroom-within-school 
is larger for each outcome (0.37 for math, 0.38 for ELA, 0.14 for growth mindset, 0.04 for 
self-efficacy, 0.05 for self-management, and 0.05 for social awareness) than the magnitude 
of differences across-school (respectively, 0.12, 0.09, 0.06, 0.03, 0.02, and 0.03). However, 
the proportion of the variance that is due to across-classroom-within-school differences is 
much smaller in SEL than in math or ELA. Conversely, the proportion of the variance that 
is due to within-classroom (i.e., across-student) differences is larger in SEL than in math or 
ELA. Among the SEL constructs, growth mindset stands out as having comparatively large 
across-classroom-within-school differences. Taken together, these results suggest that (1) 
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a greater proportion of variance is explained at the across-classroom-within-school level 
than the across-school level for each outcome, (2) we are more likely to see a very wide 
range of SEL outcomes within classrooms (i.€., across students) in a given school than 
we are to see a very wide range of math or ELA outcomes, and (3) growth mindset has a 
higher proportion of variance explained at the across-classroom-within-school level than 
the other SEL constructs. 


Table 1. Variance Decomposition of Grade 5 Models: Proportion of Variance of Posttest 
as Outcome 


Across-Classroom- 


Outcome Across-School Within-School Miidaliaba Glee icerelan 
Math O12 OrS7, OS 
ELA 0.09 0.38 0.53 
Growth Mindset 0.06 0.14 0.80 
Self-Efficacy 0.03 0.04 0.93 
Self-Management 0.02 OOS O98 
Social Awareness 0.03 0.05 0.92 


Finding 2. Classrooms in our sample serve students with different starting points 
in SEL within schools, which suggests it is important to control for those 
differences in order to measure the impact of classrooms on students’ SEL 
over the course of a school year 


We repeated the same variance decomposition analysis for the scale scores of 
students from the prior year (regardless of their prior school or classroom) in order to 
quantify differences in students’ “starting points"—in other words, to examine whether 
students in some classrooms have higher prior SEL than students in other classrooms at 
the beginning of the school year. If they do, then we would need to take those differing 
Starting points into account in order to assess how much students in a given classroom 
are growing or improving over the course of the school year. 
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Table 2. Variance Decomposition of Grade 5 Models: Proportion of Variance of Pretest as 
Outcome 


Across-Classroom- 


Outcome Across-School Within=School iid aliaba@it-scotelan 
Math Os O37. Ovo 
ELA 0.12 OBS OSS 
Growth Mindset 0.08 0.09 OSS 
Self-Efficacy 0.03 0.06 0.91 
Self-Management 0.04 0.07 0.89 
Social Awareness 0.03 0.03 0.94 


As Table 2 shows, there are similar proportions of variance explained at each of the 
three levels in the pretest as there is in the posttest (see Table 1). This suggests there are 
differences in students’ prior SEL and prior academic achievement at both the school and 
classroom levels. Prior work (e.g., Fricke et al., 2019; Loeb et al., 2019), which measured 
the variance of prior academic and SEL outcomes across schools, already established that 
different schools serve students with different prior math and ELA achievement and, to a 
esser degree, with different prior SEL. Table 2 extends this prior work and draws a similar 
conclusion with regard to across-classroom-within-school variance: classrooms within 
the schools in our sample are serving students with different starting points in math and 
ELA achievement and, to a lesser degree, in SEL. This motivates the need to take these 
differing starting points into account when measuring how much of an impact a student's 
classroom has on their growth in SEL over the course of a school year. 


Finding 3. We can estimate classroom-level growth models to control for different 
starting points in students’ SEL, but these models have less predictive power 
than similar models of academic achievement, and as a result, may not 
control for starting points as well 


Given that students differ in terms of their prior year SEL within classrooms, we 
estimated a value-added model using the same specification described in Loeb et al. 
(2018) and Fricke et al. (2019), except that we have included fixed effects for classrooms, 
rather than for schools. As Table 3 shows, we found that the goodness-of-fit (as measured 
by within-classroom R2, a version of R* computed using only within-classroom variance 
in the outcome and prediction) is quite low for the SEL models compared to the math 
and ELA models. This indicates that the SEL models have lower predictive power, which 
suggests (though does not conclusively determine) that the value-added model may 
not be controlling for all relevant differences in student characteristics. An alternative 
possibility is that the errors in measuring SEL are larger than what Is implied by the method 
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used to measure response error (Cronbach's alpha). This presents a challenge for the 
interpretation of the classroom effects from the SEL models, and thus warrants general 
caution against overinterpreting the results from these models. 


Table 3. Goodness-of-Fit for Grade 5 Classroom Growth Models 


(OU cere) aar=) Within-Classroom r2 


Math 0.70 
ELA 0.68 
Growth Mindset 0.19 
Self-Efficacy 0.20 
Self-Management 0.24 
Social Awareness 0.16 


Finding 4. The variance of classroom-level impacts on students’ SEL outcomes appears 
similar in magnitude to impacts on students’ math and ELA outcomes, but it 
is asmaller proportion of the total variance of student growth compared to 
math or ELA 


In order to assess how much of this impact comes from classrooms specifically, 
rather than schools more generally, we used the same variance decomposition framework 
described earlier (see Finding 1 and Finding 2), but this time applied to the student-level 
residuals extracted from the value-added model—in other words, applied to growth in 
each outcome rather than to the pretest or the posttest of the outcome. These residuals 
include both the classroom value-added effect as well as the student-level residual from 
the value-added regression in order to measure the total part of the outcome that cannot 
be explained with prior outcomes or demographics. Table 4 displays the results from this 
analysis; the sum of the variances in each row of the table (shown in the column furthest 
to the right) is the total variance of the residuals in the growth model. Due to the lower 
predictive power of the models that have SEL measures as outcomes, the variance of 
the residuals in the models with SEL outcomes is much higher than the variance of the 
residuals in the math and ELA models. In other words, outcomes are less consistent from 
year to year in the SEL measures than in the academic measures, so there is more student 
level “growth” left over to explain (this is evident in the higher variance estimates for the 
SEL measures than the academic measures in the “Sum” column of Table 4). As a result, 
the variance in student growth that Is attributable to classrooms is similar in magnitude 
for all six outcomes as shown by the variance estimates in the third column, “Across- 
Classroom-Within-School;” however, the classroom-level variance is a smaller proportion 
of the total variance than in math or ELA, as shown by the percent in parentheses in that 
column. 
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Table 4. Variance Decomposition of Grade 5 Classroom Growth Models: Variance (and 
Percent) of Growth as Outcome 


Social Awareness 0.02 (2% 0.05 (5% 


Outcome Scns. Warsniceiook Merson 
Math OO2177, 0.05 (17%) O 207774) 0.28 (100%) 
ELA 0.01 (4% 0.03 (10%) 0.24 (86% 0.28 (100%) 
Growth Mindset 0.02 (3% 0.07 (9%) 0.69 (88% 0.78 (100%) 
Self-Efficacy 0.02 (2% 0.05 (6%) 0.77 (92%) 0.84 (100%) 
Self-Management 0.01 (1%) 104757) 0.74 (94% 0.79 (100%) 
(5%) 


0.82 (93% 0.89 (100%) 


Finding 5. Classrooms that have a large impact on students’ growth in math and ELA 
are not necessarily the same classrooms that have a large impact on their 
growth in SEL (or vice versa) 


We examined the associations between classroom effects on the six different 
outcomes to see if classrooms with impacts in some domains tend to have similar impacts 
on other domains. Figure 1 shows the correlation between the classroom effects for each 
pair of outcomes; for example, the top left box shows the correlation between classroom 
effects on social awareness and classroom effects on math (r = 0.07, with a standard 
error of 0.02). Boxes that are darker red show little to no correlation, whereas boxes 
that are light orange or yellow show stronger correlations. The strongest relationship is 
between growth in the two academic subjects, math and ELA (r = 0.62). Growth in social 
awareness, self-management, and self-efficacy are all strongly related to one another as 
well (0.43 < r< 0.52). Growth mindset stands apart from other SEL constructs, because it 
is the SEL construct most strongly correlated with math and ELA, and is the most weakly 
correlated with the other three SEL constructs. These results indicate that classrooms 
having a large impact on students’ growth in math and ELA are not necessarily the same 
classrooms that have a large impact on their growth in SEL (or vice versa). In addition, 
there are some classrooms that have a large impact on students’ growth mindset that also 
have an impact on students’ academic achievement; however, as we discuss in the longer 
working paper, this may be an artifact of the poorer psychometric properties of the growth 
mindset construct compared to the other SEL constructs. 
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Figure 1. Weighted Correlations Among Classroom Effects on Each Outcome) 
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Conclusion and Implications 


This brief summarizes five key findings emerging from our preliminary analysis 
of whether classroom-level value-added models can measure classroom impacts on 
students’ self-reported SEL. Using the same approach that has been used to estimate 
school-level impacts on SEL (Fricke et al., 2019; Loeb et al., 2019), we find that classrooms 
differ with respect to their end-of-year self-reported SEL, their prior-year self-reported 
SEL, and their growth in SEL—even after accounting for school-level impacts. Although 
classroom effects are similar in magnitude among the four SEL outcomes measured 
(growth mindset, self-efficacy, self-management, and social awareness) and the academic 
outcomes (math and ELA), the classroom effects for SEL are proportionally smaller relative 
to the classroom effects for academics. In addition, the lower explanatory power (PR) of 
the SEL models relative to the academic models means it is less clear that these are causal 
ffects that have appropriately controlled for student-level characteristics. Although we 
recommend interpreting the results presented here with caution given the lower R? of the 
SEL models, these results nonetheless indicate there might be measurable student growth 
in SEL that is impacted in some way by the environment of the classroom and the school. 
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As conversations around appropriate and valid uses of measures of students’ SEL 
continue, this brief aims to provide some preliminary evidence as to whether we can 
reliably measure classroom-level impact on students’ SEL using the CORE self-report 
SEL surveys. In doing so, we aim to contribute to the growing body of knowledge about 
appropriate and innovative uses of measures of students’ non-cognitive and social- 
emotional learning. In addition, these results set the stage for additional research 
investigating how and whether classroom-level interventions and support might causally 
produce measurable impacts on students’ SEL, which further informs how schools and 
districts think about programs, policies, and initiatives to improve students’ SEL. 
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